Large-scale computing systems, such as those associated with network-based production services, have become widely available in recent years. Examples of such systems include online merchants, internet service providers, online businesses such as photo processing services, corporate networks, cloud computing services, web-based hosting services, etc. These entities may maintain large numbers of computing devices (e.g., thousands of hosts) which may be hosted in geographically separate locations and which may be configured to process large quantities (e.g., millions) of client requests daily or even hourly. Ensuring that these services can scale to handle abnormally high loads is a non-trivial problem.
Existing test solutions are frequently not scalable enough to handle storing, accessing, processing and/or applying a load to test at the size of today's large production systems. As a further complication, it may be desirable to test for some time periods having loads that are many times the load of other time periods. For example, a business may want to test how a network site will handle increased traffic during a time period for which the business is advertising a special promotion, or test how a retail website will handle a volume of traffic expected on peak shopping days (e.g., Black Friday or Cyber Monday).
Consequently, traditional load generators frequently need to send large (sometimes very large) amounts of data to the service under test. Traditional testing platforms frequently do not provide a clear difference between the data preparation and the actual transaction execution on the service under test and therefore the collection of latency metrics (and/or other performance metrics) may be skewed. In other words, if the data preparation is slow, but the actual transaction is fast, the slowness of the data preparation may skew collected performance metrics. Similarly, errors performance encountered during data preparation may (such as file I/O errors, or data validation errors) may count towards the number of performance errors that are reported for the service under tests.
Existing testing platforms directed to testing network-based production services frequently utilize programming libraries or frameworks that provide individual methods/class which the developers call from within their application to create a load generator that will test the production services. For example, some existing testing development frameworks may provide Java classes as building blocks that users/programmers combine into their own program. For example, programmers may declare a dependency on the Java JAR that contains the load generation code, and then piece the objects together. There may be a Java class responsible for generating transactions at a specific number of transactions per second, which programmers may use directly. This approach may create a very tight (and restrictive) contract between the programmer building the load generator and the load generation framework. Programmers generally need low-level knowledge of the way the framework works, which may hinder the flexibility of framework. Often innovation within the framework may only occur by introducing changes to the load generation framework API, possibly breaking existing applications.
Other existing frameworks may require the programmer to implement a very specific interface in order to interact with the framework. Such an interface frequently is not scalable due to having a “one-size-fits-all” approach. Additionally, existing interfaces often require programmers to create a more complex application than needed. For example, the interface may have to support diverse testing scenarios that may have little in common with each other and that may not be required by any one particular application.
While embodiments are described herein by way of example for several embodiments and illustrative drawings, those skilled in the art will recognize that embodiments are not limited to the embodiments or drawings described. It should be understood, that the drawings and detailed description thereto are not intended to limit embodiments to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope as defined by the appended examples. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description or the examples. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning “having the potential to”), rather than the mandatory sense (i.e., meaning “must”). Similarly, the words “include,” “including,” and “includes” mean “including, but not limited to.”